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Abstract. The Mock LISA Data Challenges (MLDCs) have the dual purpose 
of fostering the development of LISA data analysis tools and capabilities, and 
demonstrating the technical readiness already achieved by the gravitational-wave 
community in distilling a rich science payoff from the LISA data output. The first round 
of MLDCs has just been completed: nine challenges consisting of data sets containing 
simulated gravitational wave signals produced either by galactic binaries or massive 
black hole binaries embedded in simulated LISA instrumental noise were released in 
June 2006 with deadline for submission of results at the beginning of December 2006. 
Ten groups have participated in this first round of challenges. All of the challenges had 
at least one entry which successfully characterized the signal to better than 95% when 
assessed via a correlation with phasing ambiguities accounted for. Here we describe 
the challenges, summarise the results, and provide a first critical assessment of the 
entries. 



1. Introduction 

At the LISA International Science Team (LIST) meeting of December 2005, the Working 
Group on Data Analysis (LIST- WG IB) decided to organise several rounds of MLDCs 
with the dual purposes of (i) fostering the development of LISA data analysis tools 
and capabilities, and (ii) determining the technical readiness already achieved by 
the gravitational wave community for distilling a rich science payoff from the LISA 
data output. These challenges are meant to be blind tests, but not contests. The 
intent is to encourage the quantitative comparison of results, analysis methods, and 
implementations. 

A MLDC Task Force was constituted at the beginning of 2006 and has been working 
since then to formulate challenge problems, develop standard models of the LISA mission 
and gravitational wave (GW) sources, provide computing tools (e.g. LISA response 
simulators, and source waveform generators), establish criteria for the evaluation of the 
analyses, and provide any technical support necessary to the challenge participants. 
The first round of challenges involve the distribution of several data sets, encoded in a 
simple standard format, and containing combinations of realistic simulated LISA noise 
with the signals from one or more GW sources with parameters which were unknown 
to the participants. The participants were then asked to return the maximum amount 
of information about the sources and to produce technical notes detailing their work. 

The release of the first round of challenge data sets was announced in June, 2006 at 
the Sixth LISA International Symposium hosted by the Goddard Space Flight Center in 
Greenbelt, Maryland [HE]. John Baker (a member of the MLDC Task Force who did not 
participate in the first round) was appointed as MLDC1 coordinator. The coordinator 
was responsible for generating the challenge data sets, receiving the results from the 
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Table 1. Groups that participated in the Mock LISA Data Challenge 1. The challenges 
for which each group submitted results are marked by •. 



Group 



AEI 

Ames 

APC 

Goddard 

GLIG 

Krolak 

JPL/Caltech 

MT /AEI 

MT/JPL 

UTB 



Galactic Binaries Massive 
Single Source Multiple Sources Black Holes 



1.1.1a 1.1.1b 1.1.1c 1.1.2 1.1.3 1.1.4 1.1.5 1.2.1 1.2.2 



participants, and posting both the key data files and results as soon as possible after 
the submission deadline of December 4, 2006. 

The challenge data sets include a total of 9 year-long data sets which are described in 
detail on the MLDC website [3], the Task Force wiki [4], and the Omnibus document for 
Challenge 1 The challenge data sets are broadly grouped into three categories: (1.1) 
white dwarf binaries (WDs), (1.2) supermassive black holes (SMBHs), and (1.3) extreme 
mass ratio inspirals (EMRIs). The problem of detection of EMRIs is considered more 
difficult than the others, so the deadline for submission of results for the 1.3 challenges 
is extended to June, 2007. Consequently, in this paper we will discuss the results of 
challenges 1.1 and 1.2. 

The WD challenges consist of three single source data sets with the GW frequency 
around 1 mHz (1.1.1a), 3 mHz (1.1.1b) and 10 mHz (1.1.1c) and four multiple source 
data sets with isolated sources of known (1.1.2) and unknown (1.1.3) sky locations and 
frequencies, and overlapping sources with a low (1.1.4) and high (1.1.5) density of sources 
in frequency space. The SMBH challenges consist of two single source data sets. In one 
(1.2.1) the SMBH binary merges during the observation time, and in the other (1.2.2) 
the merger takes place between one and three months after the end of the data set. 

2. Overview of MLDC1 Submissions 

Ten groups submitted results for MLDC1 by the deadline. These results have been 
posted on the MLDC website. They include the technical notes submitted by the 
challenge participants and the files with the "best parameter fits" for the data sets. 
Table [1] provides a summary of the groups and their submissions for MLDC1. 

With the exception of Challenges 1.1.5 and 1.2.2, every Challenge data set was 
analysed by at least three groups. Here we briefly summarise the approaches used by 



MLDC first round report 



4 



each group. More detailed descriptions from many of the groups can be found elsewhere 
in these proceedings [SI El El El ECU EE2J EES! EEH EES! EES] or in the technical notes on 
the MLDC web page [3]. Several groups used variations on matched filtering methods 
on many of the challenges. The Ames group at the NASA Ames Research Center 
employed a user-refined grid search on a number of the WD challenges. The AEI 
group and Andrzej Krolak both used grid-based methods. The Global LISA Inference 
Group (GLIG), Montana- JPL, and Montana-AEI groups employed variations on Markov 
Chain Monte Carlo (MCMC) methods [TF]. The JPL-Caltech group used a multi-stage 
approach that combined time-frequency methods with grid-based and MCMC searches. 
The Montana- JPL group also used a genetic algorithm [18]. The APC group has 
also implemented an hierarchical approach which first matches the annual amplitude 
modulation and then follows with a full matched filtering. Two groups did not use 
matched filtering at all. The UTB group used a tomographic search that employed the 
Radon transform [19] while the Goddard group at the NASA Goddard Space Flight 
Center developed a time- frequency method that uses the Hilbert-Huang transform [20J . 
Although some of these groups have well-developed and mature algorithms, most groups 
are currently in various stages of development and so many of the entries are incomplete 
or suffer from known bugs which could not be hunted down before the December 
deadline. Some of the algorithms that are under development are meant to be part 
of an hierarchical search and so they only return a subset of the parameters needed to 
fully characterise the source. 



3. Assessment 



The wide variety of approaches and maturity of the algorithms makes it difficult to 
develop a single assessment that can adequately compare all entries. For those entries 
that have returned enough parameters to sufficiently generate a recovered waveform, we 
can compare the recovered waveform h rec , with the waveform generated from the "true" 
parameters h^ ey , using: 

^-^2 (^"key ^recl^key ^rec) ^ ^ 

where (*|*) is the noise (S n ) weighted inner product summed over channels (i), defined 
by: 



/max (yQ-ibi + Ojbi 



(a\b) = 2 / ±- Uj (2) 

J /min n 

and D is the dimension of the parameter space used to generate the templates. We 
realise that the A%2 is not a perfect figure of merit as, for example, it does not account 
for deduced uncertainties in the recovered parameters. It is however easy to compute 
and is quite sufficient to indicate whether the recovered parameters differ greatly from 
those used for the key waveform. The channels used are the noise orthogonal pseudo A 
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and E channels [21J: 

A = (2X - Y - Z) /3, E=(Z- Y) /V3, (3) 

and X, Y, and Z are the standard TDI variables. We can also compute the signal-to- 
noise ratio (SNR) for both /ik ey and h Tec using: 

SNR = _P= (4) 

and compare the recovered SNR with the key SNR. Finally, we calculate the correlation 
between h^ ey and h Iec with: 

q _ (^-kcy l^-rcc) 

\J (^key | ^-kcy ) (^roc | ^tcc) 

Some groups reported a known ambiguity in the initial phase and polarisation angles 
with results being given modulo 7r/2 or 7r. Obviously, a difference of tt in the initial 
phase can significantly degrade the performance of an entry as calculated using Ax 2 , 
SNR, or C. Consequently, we have also computed these measures with the initial phase 
shifted by either tt or tt/2 as necessary. 

Another measure of the success of a given algorithm is the accuracy with which 
it returns specific parameters. This approach allows us to also evaluate those entries 
which do not return enough parameters to generate h Tec . For each parameter Aj, we can 
determine the difference between the key parameter and the recovered parameter using: 

AA = Akey — A rec . (6) 

We note that it is not necessarily appropriate to use the Fisher Information Matrix 
(FIM) to determine the quality of parameter recovery. If the algorithm settles on a 
secondary maximum of the likelihood function or there are other systematic errors, 
then the results can be far from the regime of validity for the FIM approximation to 
expected errors in parameter estimation. For more details, see Vallisneri's review of the 
FIM [22]. 

The white dwarf binary challenges required the recovery of 7 parameters to fully 
characterise each source. These parameters are: the amplitude A, the frequency /, 
they sky location 9, <p, the angle of inclination t, the polarisation angle ip, and the 
initial phase 4>q. In Table |2j we list the values of the measures for each challenge entry 
for challenges 1.1.1 — with the exception of the UTB entry. Since the UTB algorithm 
only returns frequency (in intervals of resolvable frequency bins ~ 32 nHz) and sky 
position, it cannot be included in this comparison. However, it can be included in the 
comparison of parameter differences given in Table [3j It should be noted that the GLIG 
cluster crashed before the completion of the algorithm and therefore the MCMC chain 
did not have the chance to burn in to the final values. We also note that although the 
measures of Challenge 1.1.1c in Table [2] seem to be quite bad, it is important to note 
that the accuracy in recovery of the sky positions and frequencies is still comparable to 
Challenges 1.1.1a and 1.1.1b. 
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Table 2. The performance of challenge entries on the single binary challenges as 
calculated using A% 2 , SNR, and C. The correction of the initial phase by a factor of 
7r or 7r/2 is indicated by an asterisk (*). 



Group 


AX 2 


SNR 




Challenge 1.1.1a (SNR kcy 


CI 1 Q7\ 

— ol. lo/ ) 




AEI 


8.095 


50.604 


0.989 


Ames 


7.155 


51.032 


0.997 


APC 


423.406 


-8.007 ■ 


-0.135 


APC* 


229.115 


50.385 


0.990 


Krolak 


778.888 


0.933 - 


-0.004 


Krolak* 


1.036 


1 .uOO 


n QQQ 


MT/JPL (BAM) 


1.902 


51.178 


0.998 


MT/JPL (GA) 


1.796 


51.138 


0.998 


Challenge 1.1.1b (SNR kcy 


= 37.251) 




AEI 


47.913 


33.104 


0.874 


Ames 


64.371 


32.067 


0.822 


Krolak 


841.074 


-37.038 - 


-0.996 


Krolak* 


2.566 


37.038 


0.996 


MT/JPL (BAM) 


7.735 


36.856 


0.980 


MT/JPL (GA) 


8.371 


36.808 


0.979 


Challenge 1.1.1c (SNR koy = 


= 101.390) 


AEI 


2399.307 


-14.373 - 


-0.144 


GLIG 


1788.991 


14.496 


0.142 


Krolak 


5997.595 


-98.126 - 


-0.968 


Krolak* 


97.603 


98.126 


0.968 


MT/JPL (BAM) 


945.541 


63.383 


0.623 


MT/JPL (GA) 


1376.143 


43.564 


0.424 



The multi-source challenges present a different problem for assessment, since there 
is the possibility of false positives and false negatives. Consider the possibility in which 
the recovered parameters for one binary out of many are wildly off. If one were to use 
the correlation between each recovered template and one of the source binaries, it is 
possible to count the recovered binary as a false positive and the true binary as a false 
negative. Given the phasing issues that were apparent in challenges 1.1.1, it is quite 
likely that there will be several false positive/false negative pairs if such a correlation 
analysis is used. Consequently, we determine which recovered template goes with which 
source by looking for template / source pairs that are within one resolvable frequency bin 
of each other. The overall success of the recovery can be measured using the combined 
signal of the entire population of recovered binaries as h TCC and comparing with the entire 
population of the true source binaries as hk ey . In this case, we note that the dimension 
D of the recovered parameter space depends upon the number of recovered sources. 

The challenge 1.1.2 data set contained 20 "verification" binaries whose frequencies 
and sky location were given to the participants. Six of these binaries were taken from 
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Table 3. The performance of challenge entries on the single binary challenges as 
calculated using recovered parameter differences. 



Group A/ (nHz) 


ZAP 




A In A 
Z_i 111 1\ 


LAI 




Ash 

la po 


Challenj 


z,c 1.1.1a 














AFT 




-0.018 


0.001 


-0.078 


-0.101 


0.157 


-0.065 


Ames 


-1.889 


-1.159 


3.127 


0.337 


0.503 


0.181 


-0.126 


APC 


1.343 


-0.030 


-0.011 


0.807 


0.217 


0.174 


1.202 


Krolak 


0.980 


0.028 


-0.008 


0.113 


0.180 


0.208 


-2.089 


MT/JPL (BAM) 


-1.367 


-0.015 


-0.008 


-0.046 


-0.084 


0.196 


-0.228 


MT/JPL (GA) 


-1.044 


n niQ 


-U.UUo 


fl 077 
-U.U / / 


-u.uyi 


n 00/i 


-U.oUo 


UTB 


-3.209 


0.143 


0.603 










Challen; 


re 1.1.1b 














AEI 


0.399 


-0.049 


0.001 


-0.009 


-0.045 


0.020 


0.432 


Ames 


21.098 


-0.606 


0.004 


0.171 


0.048 


0.028 


2.173 


Krolak 


0.341 


0.037 


-0.004 


-0.112 


-0.042 


-0.042 


-3.098 


MT/JPL (BAM) 


0.434 


-0.040 


0.003 


-0.025 


-0.042 


0.029 


0.097 


MT/JPL (GA) 


0.314 


-u.uoy 


U.UUO 


fl D44 


- U .U^:^: 


u.uou 


n 1 1 7 


UTB 


-4.299 


0.198 


0.007 










Challen; 


ge 1.1.1c 














AEI 


-0.405 


0.012 


-0.001 


0.312 


-0.159 


0.127 


1.501 


GLIG 154.850 


0.306 


0.178 


0.341 


0.939 


0.722 


2.413 


Krolak 


-5.210 


0.059 


-0.010 


-0.194 


-0.268 


0.451 


2.747 


MT/JPL (BAM) 


-0.330 


0.008 


-0.001 


0.309 


0.033 


-0.609 


2.148 


MT/JPL (GA) 


0.311 


0.013 


-0.001 


0.652 


-1.062 


-0.614 


2.026 


UTB 


8.577 


0.139 


0.066 











the list of known binaries available on Gijs Nelemans website [23] and the remaining 14 
were simulated binaries. Of the four groups that submitted entries for challenge 1.1.2, 
two of them did not use the additional information of sky location and frequency in their 
searches. The Montana/ JPL group used the same search algorithms as they used in all 
of the 1.1 challenges, and the UTB group used their tomographic algorithm to obtain the 
frequency and sky location. The UTB group successfully recovered 17 of the 20 source 
binaries. The other three groups successfully recovered all 20 source binaries, although 
the AEI group had very low correlation with the 2 highest frequency binaries. This 
is understandable since the AEI group used the low frequency approximation (which 
is valid for frequencies below about 3 mHz) for calculating their templates and the 2 
highest frequency binaries were at frequencies above 6 mHz. These two binaries are also 
responsible for most of the loss in correlation for this entry. The performance of the 
three groups that returned the complete parameterisation of each binary recovered is 
given in Table H] 

Challenge 1.1.3 also contained 20 binaries isolated in frequency space between 0.5 
mHz and 10 mHz. In this challenge, all binaries were drawn from the simulation and all 
parameters were blind. Five groups submitted entries, although the UTB group again 
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Table 4. The performance of challenge entries on the verification binary challenge 
1.1.2 as calculated using A% 2 , SNR, and C. Since every group returned the full 20 
binaries, the dimension of the model is D = 140. The SNR of the key is 544.952. 



Group 


Ax 2 


SNR 


C 


AEI 


1443.59 


339.262 


0.624 


Ames 


227.007 


516.471 


0.948 


MT/JPL (BAM) 


19.02 


544.165 


0.998 


MT/JPL (GA) 


194.46 


519.712 


0.954 



Table 5. The performance of challenge entries on the isolated binary challenge 1.1.3 as 
calculated using A^ 2 , SNR, and C. Since not all groups returned the full 20 binaries, 
the dimension of the model is calculated by D = 7 x iV" rec , where A rcc is the number of 
recovered binaries. The SNR of the key is 122.864. The correction of the initial phase 
by a factor of tt or ir/2 is indicated by an asterisk (*). 



Group 


Ax 2 


SNR 


C 


N 

1 v rcc 


AEI 


67.67 


89.789 


0.726 


16 


Ames 


48.42 


104.236 


0.841 


13 


Krolak 


204.36 


9.558 


0.080 


20 


Krolak* 


150.95 


38.770 


0.323 


20 


MT/JPL (BAM) 


44.81 


98.838 


0.797 


19 


MT/JPL* (BAM) 


4.16 


121.64 


0.981 


19 


MT/JPL (GA) 


19.66 


113.797 


0.914 


18 



only provided frequency and sky location. The UTB group successfully identified 14 of 
the 20 binaries. The performance of the other four entries are given in Table [5j The sky 
locations returned by each group are compared with the source positions in Figure [TJ 

The remaining two white dwarf challenges each contained approximately 45 
overlapping sources. Challenge 1.1.4 contained 45 sources in a 15yuHz band starting at 3 
mHz, while Challenge 1.1.5 contained 33 sources in a ±1.5/iHz band centered on 3 mHz. 
Challenge 1.1.4 had an average density of 0.095 sources per resolvable frequency bin, 
and Challenge 1.1.5 had an average density of 0.35 sources per bin. These Challenges 
were actually more difficult than these source densities might indicate since in both 
cases, there were at least 3 frequency bins which contained at least 2 binaries each. 
There was one case in Challenge 1.1.5 with 3 binaries in one frequency bin. Despite 
this additional complication, both groups managed to recover a respectable number of 
sources that matched with binaries in the key. The performance of the two groups that 
submitted complete parameter sets for each binary recovered are listed in Table [61 

The supermassive black hole challenges required recovery of 9 parameters describing 
the source: the chirp mass A4, the reduced mass /i, the luminosity distance D L , the 
time of coalescence t c , the sky location 6, <p, the initial angle of inclination t, the 
inital polarisation angle if), and the initial orbital phase 0o- As with the white dwarf 
challenges, the quality of the recovered signal can be described by the measures given 
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Figure 1. Recovered sky positions from each entry for Challenge 1.1.3. The source 
positions are indicated by + and the recovered positions are indicated by o. Each plot 
includes only those sources which are within a frequency bin of a recovered source. 
The plots are of ecliptic latitude vs. longitude in degrees. 



Table 6. The performance of challenge entries on the overlapping binary challenges 
1.1.4 and 1.1.5 as calculated using A% 2 , SNR, and C. Since not all groups returned 
the full number of binaries, the dimension of the model is calculated by D = 7 x ^V rec , 
where A^ roc is the number of recovered binaries. A^ mat ch is the number of recovered 
binaries that matched in frequency with a binary in the key. 



Group A X 2 SNR 


C N rcc 


■^match 


Challenge 1.1.4 (SNR koy 


= 201.129) 




AEI 85.63 159.893 
MT/JPL (BAM) 6.19 197.828 


0.792 26 
0.976 43 


16 
39 


Challenge 1.1.5 (SNR koy 


= 178.261) 




AEI 519.21 116.822 
MT/JPL (BAM) 11.96 172.582 


0.654 5 
0.963 27 


4 
23 
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in Equations [T] and HI In place of the correlation, we compute several overlaps using: 

o a = , = ( 7 ) 



^(KejKey) (^ecl^ec) 



where a denotes the particular TDI channel being used. In order to mitigate the effects 
of a possible error in the initial phase, we have also computed Ox, maximised over the 
phase: 



max* (O x ) = V(^cl^ ey (0o = 0))' + (^ c |^ ey (0o = vr/2)) z . (8) 

There were two groups that returned a full characterisation of the signal for Challenge 
1.2.1. The Montana/AEI group had a constant phase difference, and when this phase 
is corrected, the performance of both the JPL/Caltech and Montana/AEI groups is 
quite good. The Goddard group is developing a new algorithm using the Hilbert-Huang 
Transform that is in a very preliminary stage and has only returned M. and t c for this 
challenge. Because of a known secondary maximum in sky location, we also check the 
antipodal sky position: 

9^-9 , 0^0±tt. (9) 

However, this adjustment also requires a change in the values of the inclination and 
polarisation angles as well. This is accomplished by substituting the initial returned 
values of 9, 0, l, and if) into: 

cos i = cos 9 sin 9 cos (0 — $) + cos 6 sin 9 (10) 
sin 9 cos (0 — $) sin — cos 9 cos 9 

tan ^ = n • 7T ^ >, (11) 

sin B sin (0 — <3>J 

and solve these equations for the orientation angles of the orbital angular momentum 
vector, (0,$). Once we have the values of (B, $), we then use these values and the 
antipodal sky position from Equations M in the above equations [TD] and [TT] to determine 
the new values of l and ip. Once these have been found, the new values can be used to 
provide a more realistic estimate of the error in the returned values. We have applied this 
transformation to the JPL/Caltech entry and also adjusted the polarisation phase for 
the Montana/AEI entry in Challenge 1.2.1. Only one group (Montana/AEI) submitted 
an entry for Challenge 1.2.2. The measures for each submission under both of these 
challenges are given in Table [71 We have also determined the errors in the recovered 
parameters for all entries in Challenges 1.2.1 and 1.2.2. These are presented in Table El 



4. Conclusions 

The first round of the Mock LISA Data Challenges successfully attracted over ten groups 
to work on the problem of LISA data analysis. These groups attacked several of the 
challenges with a variety of different approaches. The algorithms and codes used in 
the challenges were at different levels of maturity and completeness of the pipelines. 
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Table 7. The performance of challenge entries on the supermassive black hole binary 
challenges 1.2.1 and 1.2.2. The results after a correction of the initial phase are 
indicated by a *. 



Group 


Ax 2 


SNR O a 


O e 


max0 o (O x ) 




Challenge 


1.2.1 (SNR kcy = 


667.734) 




JPL 


261.48 


664.47 0.994 


0.996 


0.9955 


MT/AEI 


10289.29 


524.29 0.790 


0.791 


0.9998 


MT/AEI* 


105.50 


662.87 0.998 


0.998 


0.9998 




Challenge 


1.2.2 (SNR key = 


= 104.19) 




MT/AEI 


1.41 


104.29 0.997 


0.998 


0.9955 



Table 8. The performance of challenge entries on the supermassive black hole binary 
challenges 1.2.1 and 1.2.2 on the estimation of recovered parameters. The angles are 





absolute 


errors, all other errors are relative. 








Group 


AM/M 

(xlO" 4 ) 


An/n AD L /D L At c /t c AO A0 
(xl0~ 6 ) 


At 


Aip 


A0 O 


Challenge 1.2.1 (Reported values) 


JPL 

MT/AEI 
Goddard 


7.35 
0.98 
434.00 


0.011 1.101 3.35 1.030 -3.170 
0.001 0.042 0.26 0.001 0.001 
113.00 


1.32 
0.02 


-2.65 
3.14 


0.004 
0.004 


Challenge 1.2.1 (Angle adjusted values) 


JPL 

MT/AEI 


7.35 
0.98 


0.011 1.101 3.35 -0.043 -0.032 
0.001 0.042 0.26 0.001 0.001 


-0.58 
0.02 


-0.31 
-0.00 


0.004 
0.004 


Challenge 1.2.2 


MT/AEI 


3.09 


0.037 0.273 182.00 0.019 0.005 


-0.71 


-2.16 


-0.002 



Nonetheless, all challenges had at least one entry which successfully characterized the 
signal to better than 95% when assessed via the correlation with phasing ambiguities 
accounted for. In the overlapping source challenge 1.1.5, one group was able to recover 
true binaries at a source density of ~ 0.25. Most groups also discovered small bugs or 
discrepancies in definitions of some of the parameters used to characterise the signal. 
The first round entries were a success, especially considering that most groups had less 
than 5 months to work out interfacing issues and assess their codes. Those groups that 
participated in round 1 have begun implementing the lessons learned for use in round 
2 [23], which is another successful outcome of the challenges. The MLDC Task Force 
has also begun to address the issues that have been raised as regards assessment of the 
entries. As the assessments become more refined in future challenges, we anticipate 
developing assessments for the true LISA data for which there is no key file. 
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